Less is More: Facial Landmarks can Recognize a Spontaneous Smile

Smile veracity classification is a task of interpreting social interactions. Broadly, it distinguishes between spontaneous and posed smiles. Previous approaches used hand-engineered features from facial landmarks or considered raw smile videos in an end-to-end manner to perform smile classification tasks. Feature-based methods require intervention from human experts on feature engineering and heavy pre-processing steps. On the contrary, raw smile video inputs fed into end-to-end models bring more automation to the process with the cost of considering many redundant facial features (beyond landmark locations) that are mainly irrelevant to smile veracity classification. It remains unclear to establish discriminative features from landmarks in an end-to-end manner. We present a MeshSmileNet framework, a transformer architecture, to address the above limitations. To eliminate redundant facial features, our landmarks input is extracted from Attention Mesh, a pre-trained landmark detector. Again, to discover discriminative features, we consider the relativity and trajectory of the landmarks. For the relativity, we aggregate facial landmark that conceptually formats a curve at each frame to establish local spatial features. For the trajectory, we estimate the movements of landmark composed features across time by self-attention mechanism, which captures pairwise dependency on the trajectory of the same landmark. This idea allows us to achieve state-of-the-art performances on UVA-NEMO, BBC, MMI Facial Expression, and SPOS datasets

Tracking Topical Evolution In Large Document Collections

A large document collection that builds up over time usually contains a number of different themes. All of these themes or topics are not equally important at the same time. One topic might have high probabilities in some years due to some relevant events, and low probabilities in other years. Analyzing the evolution of such topics has useful applications in a variety of domains, for example, helping researchers to quickly see the changes of research topics in an area, assisting intelligence agents in tracking the activities of a terrorist group, or monitoring damages caused by a natural disaster. In this Dissertation, I present three different models that I developed to capture the evolution of topics in dynamic corpora in different domains. First, I present a novel algorithm for finding the lineage of a scientific article. The algorithm provides a unique way of encoding temporal information in a document that helps discovering more interesting lineage compared to the other state-of-the-art models. Then, I propose a topic model called STEM that accurately extracts high-level themes from a corpus, and also simultaneously captures the evolutionary patterns of those themes. Topic models have been used for summarizing text corpora for a long time, but STEM is the first model that combines the ideas of supervised inference and topical evolution. In many contexts - political conflicts, for instance - topics dont evolve only over time, they have different degrees of impact in different geolocations as well. Therefore, I finally developed a new spatiotemporal topic model that can track geopolitical conflicts over the temporal and geographical dimensions. For each of these models, I present results of qualitative and quantitative analysis on multiple real-world datasets demonstrating the effectiveness of the model